[CI][ROCm] Check AMD monokernels stage by stage against float64 references - #63486

• monokernels.yaml: one test area for every AMD monokernel (DeepSeek-V4.1 on MI355 and MI300, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3), each with exactly two steps: a blocking Mono I/O step and an optional [Nightly] Mono E2E step, with curated source_file_dependencies; the existing steps move here from models_basic.yaml and kernels.yaml
• mono_reference.py and test_mono_primitives.py: a shared float64 plain-torch reference and checks of every device primitive the kernels share (transcendentals, scaled MFMAs, MXFP8/MXFP4 codes and rounding)
• Per-model I/O tests drive each fused stage on synthetic weights and compare it with the float64 reference; every tolerance is derived from the formats, roundings and accumulation involved, with the derivation next to the bound, and checked against injected bugs
• Comparisons against vLLM's and AITER's own ops move to the nightly steps, which also run gsm8k on the real checkpoint with a post-eval check that the monokernel ran on every rank (final_rpc_checks, monokernel_checks.py); thresholds are 3 binomial standard errors below the mono-off accuracy
• Each gsm8k config pins its checkpoint and draft model to the revision it was calibrated on, so CI caches one snapshot per model and a checkpoint update can't silently change results
• DeepSeek-V4.1 mono tests move to tests/models/deepseek_v41/amd_specialized_decode/, next to Kimi-K3's and MiniMax-M3's
• AGENTS.md: a PR adding an AMD monokernel adds these two steps (or extends the model's), asserts accuracy rather than which code path ran, derives its tolerances, and passes 10 times in a row before submission

Monokernels landed on AMD for DeepSeek-V4.1 (#60397, #61082, #62131, #60731), Kimi-K3 (#60683), MiniMax-M3 (#62712, #62994) and DeepSeek-V4-Pro (#61482). Their CI was spread over three test areas, missed files the kernels read, compared mostly against vLLM's stock path with tolerances measured from that comparison, and could not tell whether the monokernel ran at all. Checking each stage against a slow, exact reference found two kernel bugs, each fixed in its own PR. Both are merged:

• [Bugfix][ROCm][DSv4.1] Await krdy before K2's combine reads klen in index-source launches #63576: DeepSeek-V4.1's stage_combine reads klen without waiting on krdy; a shorter stale value corrupts 3,791 of 8,704 attention-output codes and a longer one hangs the step.
• [Bugfix][ROCm][DSv4] Give zero and subnormal mono MXFP8 groups vLLM's scale code #63689: DeepSeek-V4 and V4.1 give all-zero and tiny MXFP8 groups scale code 1 where vLLM gives 0.

The architectural side is discussed on the RFC, #60904. Related open PRs: #63398 replaces MiniMax-M3's MonoKernel with ATOM's full-layer implementation, and #60833 adds a Qwen3.8 mono decode; both would get their steps here.

Hardware. Blocking steps use synthetic weights. Measured on MI355X, each fully triggered run costs about 3.3 GPU-h (DeepSeek-V4.1 14 min on 4 GPUs, DeepSeek-V4-Pro 7.5 min on 8, MiniMax-M3 7 min on 4, Kimi-K3 7 min on 8). DeepSeek-V4-Pro and Kimi-K3 need a whole mi355_8, and none of the steps fit mi355_dpx. The gsm8k runs of the four DeepSeek/Kimi configs took 2.2-3.0 GPU-h each (about 10 GPU-h per night), plus the two MiniMax-M3 configs and the nightly parity tests. New checkpoint storage per HF cache:

Validation. On 8x MI350X with the CI pins (AITER v0.1.26, FlyDSL 0.4.0), each MI355 Mono I/O step passed 10 times in a row, and each [Nightly] Mono E2E step passed once with every gsm8k config above its threshold. The MI300 steps have not run (no gfx942 here), so their derived bounds are unvalidated on hardware. Engine smokes no longer compare whole layers: each stage runs vLLM's own op on the kernels' inputs, routing and index top-k near-ties are forced identical, and every observed difference sits within one or two ulps of the derived budget. Along the way:

• AITER's tuned FlyDSL bf16 GEMM rounds split-K partials to bf16 before summing (3-5 ulp gaps in Kimi-K3's stock linears); an AITER fix is being prepared, and the engine test models it meanwhile.
• [Issue]: FlyDSL MoE stage-1 fused FP8 output uses a different MXFP8 scale rule than fused_dynamic_mx_quant_moe_sort ROCm/aiter#6548 (a coarser E8M0 scale rule in the fused FP8 MoE stage 1 at small token counts) shows up as a 1.6-1.9% gap at 3 rows; the DeepSeek-V4.1 smoke tiles vLLM's MoE call to avoid it.
• Kimi-K3's K2 tests launched each new build without the ranks first compiling it, so a slow compile could outlast the kernel's 10 s wait; every rank now compiles and meets at a barrier first, as the engine does.
• MiniMax-M3's earlier gsm8k spread did not reproduce in 24 new runs (SD 0.0066, below the binomial SE of 0.0093).

AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.